fix(server): notify watchers after atomic policy commits - #1
Open
NaveCohenMonday wants to merge 20 commits into
Open
fix(server): notify watchers after atomic policy commits#1NaveCohenMonday wants to merge 20 commits into
NaveCohenMonday wants to merge 20 commits into
Conversation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
NaveCohenMonday
force-pushed
the
2517-notify-atomic-policy-watchers/nc
branch
from
July 28, 2026 15:40
4674448 to
b9917c4
Compare
Replace yield_now() spin-loop with sleep(10ms) polling in delete_handler_ends_telemetry_for_the_resolved_sandbox_id. The single-threaded tokio runtime starves the spawn_blocking threads used by SQLite when yield_now() burns 100% CPU waiting for the delete gate entry count. Increase the timeout from 1s to 5s for consistency with similar guard tests. Signed-off-by: Derek Carr <decarr@redhat.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(examples): add supervisor middleware content guard Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(examples): refine middleware preview warning Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): add middleware policy version Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * refactor(supervisor-middleware): simplify service endpoints Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): adapt content guard to middleware enums Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): align content guard with merged middleware Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * feat(examples): add content guard smoke flow Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * chore(examples): remove smoke launcher test Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): align content guard smoke with main Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): address content guard review feedback Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * refactor(examples): parse cargo metadata with jq Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): prioritize longest content matches Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(examples): use GitHub warning alert Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(examples): merge overlapping content matches Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(examples): render preview warning on GitHub Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…#2463) The Kubernetes driver's default workspace PVC never set storageClassName, so on clusters with no default StorageClass the PVC stayed Pending and sandbox creation failed. Add a workspace_storage_class option to KubernetesComputeConfig, wired through SandboxPodParams into the generated volumeClaimTemplates. When non-empty it sets storageClassName; empty preserves the current behavior of relying on the cluster default StorageClass. Expose it via the OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS env var on both the standalone driver and the embedded gateway runtime defaults, and via the server.workspaceStorageClass Helm value. Closes NVIDIA#2442 Signed-off-by: lr90 <qiuweimin@matrixorigin.cn>
* feat(sandbox): use policy-first OCI image identity Closes NVIDIA#2331 Preserve per-field policy omission, derive Docker and Podman fallbacks from the inspected immutable image, and resolve the final numeric identity before starting agent children. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): preserve declared process identities Keep explicit policy values and OCI-declared names intact, defer passwd lookup until a primary GID is required, and refresh stale policy examples. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(supervisor): reuse resolved OCI identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(supervisor): allow Linux pre-exec arguments Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): protect resolved sandbox identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): prepare workspace for OCI identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(sandbox): own only workspace root Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): harden partial identity drops Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(sandbox): scope OCI image e2e to Docker Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): narrow OCI identity fallback scope Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(podman): cover OCI identity launch Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(podman): exercise OCI fallback in E2E Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…#2446) parse_duration_to_ms was moved to commands/common.rs since the original PR was opened, but it still split the last byte of the input with split_at(s.len() - 1), which panics when the final character is multi-byte UTF-8 (e.g. 'openshell logs my-sandbox --since 5€'). Split off the last character using its UTF-8 length instead, so invalid units surface the intended 'unknown duration unit' error. Add regression tests in commands/common.rs. Signed-off-by: Andrew White <andrewh@cdw.com>
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.5.1 to 4.5.2. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@abd2ef4...371161b) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.5.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ivers (NVIDIA#2153) * refactor(compute): extract create_sandbox_record and update_sandbox_record helpers Split apply_sandbox_update_locked into named helpers to make the two distinct paths explicit: create_sandbox_record for first-observation events and update_sandbox_record for subsequent driver snapshots on existing sandboxes. The dispatcher now uses a match on the existing record rather than an early-return guard. No behavior change. Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(compute): make sandbox readiness gateway-owned across all drivers Introduce compute_phase_components and apply_readiness_conditions to centralise the gateway's phase composition logic. The public SandboxPhase is now determined by combining the backend phase reported by the driver with supervisor session presence, independent of the driver implementation. Remove SupervisorReadiness from the driver contract. Running containers always report BackendReady; the gateway owns the Ready decision. Rename the dispatcher match to three arms so that status-less events for existing sandboxes are a documented no-op rather than a silent pass-through. Drop backend_ready_no_session and the SupervisorNotConnected condition. The BackendReady driver condition plus the Provisioning phase already communicates that the backend is up but the supervisor has not connected. The redundant condition added noise without new information. Closes NVIDIA#1951 Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(compute): expose disconnected supervisor readiness Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(sandboxes): clarify supervisor readiness lifecycle Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Bumps [actions/stale](https://github.com/actions/stale) from 10.4.0 to 11.0.0. - [Release notes](https://github.com/actions/stale/releases) - [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md) - [Commits](actions/stale@1e223db...4391f3d) --- updated-dependencies: - dependency-name: actions/stale dependency-version: 11.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…DIA#2523) The subprocess-based integration tests inherit the full parent environment. If the developer has OPENSHELL_GATEWAY_INSECURE=true set in their shell, it leaks into the spawned CLI process and causes it to connect with .with_no_client_auth(), skipping the mTLS client certificate. The test server requires mTLS, so it responds with CertificateRequired and the test fails. Strip OPENSHELL_GATEWAY_INSECURE, OPENSHELL_GATEWAY, OPENSHELL_GATEWAY_ENDPOINT, and OPENSHELL_WORKSPACE from the subprocess environment. The test already sets --gateway and --gateway-endpoint explicitly via CLI args, so these env vars should not influence the subprocess behavior. Reported-by: Seth Jennings Signed-off-by: Roland Huß <rhuss@redhat.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
ForwardSpec accepts IPv6 bind addresses, but ssh_forward_arg() emitted them unbracketed (e.g. ::1:8080:127.0.0.1:8080), which OpenSSH rejects as a bad local forwarding specification. access_url() likewise produced invalid URLs like http://::1:8080/. Extract a shared bracket_ipv6_host() helper and use it in ssh_forward_arg(), access_url() (via format_gateway_url), and format_gateway_url() so IPv6 literals are bracketed consistently. Fixes NVIDIA#2279 Signed-off-by: Russell Bryant <rbryant@redhat.com>
* test(vm): add composable QEMU test guests Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(vm): describe test VM directory structure Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(vm): replace shell catalog functions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(vm): add Fedora release guest support Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(vm): enable rootless Podman socket Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): add OCI-backed image caching Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(test-guest): accelerate cached guest startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): address review feedback Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): verify OCI cache provenance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): harden cached guest reuse Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): refresh runtime setup state Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): support E2E runner inputs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): harden runner and OCI reuse Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): prepare Podman E2E artifacts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): address review findings Signed-off-by: Drew Newberry <anewberry@nvidia.com> * revert(test-guest): remove recent Podman artifact changes Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): canonicalize scp source paths Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(test-guest): provision artifacts with Ansible Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): populate missing caches on startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(auth): implement RFC 0011 Phase 2 workspace authorization Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address PR review feedback on workspace authorization - Docker e2e: add --health-port and switch readiness probe from `openshell status` to `curl /healthz`, fixing a false-positive readiness check in OIDC mode where the CLI exited 0 without actually contacting the gateway - ListWorkspaces: move membership filtering from post-query N+1 lookups into a SQL EXISTS subquery so pagination applies to the visible set, not the global ordering. Add generic list_with_membership to the persistence layer. - Descriptor validator: reject role/scope fields on unauthenticated and sandbox auth modes, and allow-list workspace_role as user/admin and global_role as platform_admin to catch typos at startup Signed-off-by: Derek Carr <decarr@redhat.com> * fix(server): use authed request in delete telemetry test The workspace authorization added by the Phase 2 auth changes requires a Principal on every delete request. The delete-telemetry test was still using a bare Request::new, so extract_principal failed before the handler could acquire the delete gate, causing a 5-second timeout flake. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address gator review findings for workspace authorization - Inject unauthenticated-local-dev principal in no-auth gateway mode so handlers that call extract_principal() always find one. - Cap label-selector membership query at MAX_PAGE_SIZE instead of u32::MAX to bound the in-memory read. - Authorize workspace membership before resolving workspace existence in all sandbox RPCs to prevent workspace-name enumeration by non-members. - Remove dead_code allow on AuthorizedWorkspace.workspace now that callers use the normalized name from the authz result. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): close workspace-name oracle and label-selector truncation Swap authorize-before-resolve ordering in 27 handlers across provider.rs, service.rs, policy.rs, and workspace.rs to prevent CWE-203 workspace-name enumeration by non-members. Add combined membership+label SQL query (list_with_membership_and_selector) to both persistence backends so ListWorkspaces with label selectors no longer silently drops results beyond the first page of membership matches. Signed-off-by: Derek Carr <decarr@redhat.com> * test(auth): add non-member rejection and membership+label persistence tests Add comprehensive test coverage for workspace authorization changes: - Non-member rejection tests across all 44 workspace-scoped handlers (sandbox, provider, service, policy, workspace, inference) verifying PERMISSION_DENIED is returned instead of NOT_FOUND to prevent CWE-203 workspace-name oracle - Persistence test for list_with_membership_and_selector verifying SQL-level membership EXISTS + label filtering, multiple predicates, no-match cases, and pagination Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): format merged import line in sandbox tests Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address gator re-review findings on workspace authorization - Fix TUI unconditionally setting providers_v2_enabled after provider refresh; read the actual gateway setting via GetGatewayConfig at startup instead - Fix SQLite json_extract with dotted label keys (e.g. example.com/env) by quoting the key in the JSON path - Add authed_request wrappers to upstream OCI identity tests that were missing a principal after rebase - Add test proving GetGatewayConfig is accessible without Platform Admin - Add test for dotted/prefixed Kubernetes-style label key filtering Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address second gator re-review findings - Loosen GetGatewayConfig from platform_admin to scope-only so workspace users can discover providers_v2_enabled during sandbox creation with inferred-provider commands; update proto descriptor, descriptor validation, and RFC 0011 access table - Add validate_label_selector to handle_list_workspaces and escape single quotes in SQLite json_extract interpolation (CWE-89 defense-in-depth) - Re-fetch providers_v2_enabled after TUI gateway switch so the new gateway's capability is reflected - Add e2e test for workspace user with inferred-provider command - Add persistence test for adversarial label keys with SQL injection attempts - Add handler test for invalid label selector rejection in ListWorkspaces Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address third gator review findings - Cap label selector pairs at 64 (CWE-400) to bound SQLite dynamic SQL - Add SCOPE_ONLY_METHODS allowlist for scope-without-role RPCs (CWE-863) - Normalize ID-based data-plane handlers to return NOT_FOUND for unauthorized sandboxes, closing the cross-workspace oracle (CWE-203) - Fix TUI provider profile cache lookup key mismatch for legacy providers with empty profile_workspace - Add whoami to CLI skill reference command tree - Update TUI skill doc with workspace, provider, and settings coverage - Document scope/workspace orthogonality on GetGatewayConfig proto Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): extend CWE-203 normalization to policy.rs sandbox handlers GetSandboxConfig and GetSandboxLogs in policy.rs had the same fetch-before-authorize pattern that leaked cross-workspace sandbox existence. Promote fetch_and_authorize_sandbox to pub(super) and use it from both sandbox.rs and policy.rs handlers. Signed-off-by: Derek Carr <decarr@redhat.com> * test(auth): update assertions for CWE-203 sandbox ID normalization Cross-workspace sandbox access via ID-based handlers now returns NOT_FOUND instead of PERMISSION_DENIED to prevent existence inference. Update the unit test and OIDC e2e assertion to match. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): narrow CWE-203 error mapping and correct whoami output formats Only remap PERMISSION_DENIED to NOT_FOUND in fetch_and_authorize_sandbox and RevokeSshSession, letting INTERNAL and UNAUTHENTICATED propagate as-is. Fix whoami --output format values in cli-reference.md to match the actual CLI (table/json/yaml, not text/json). Signed-off-by: Derek Carr <decarr@redhat.com> * fix(ci): share network namespace with Keycloak in containerized CI In GitHub Actions job containers, Docker port publishing lands on the host, not inside the job container. Detect this environment and attach Keycloak to the job container's network namespace instead, with hardened defaults (cap-drop ALL, no-new-privileges, loopback-only listener). Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(gator): preserve resolved review feedback Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gator): make follow-up reviews pragmatic Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(gator): enforce review convergence Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(vm): add composable QEMU test guests Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(vm): describe test VM directory structure Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(vm): replace shell catalog functions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(vm): add Fedora release guest support Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(vm): enable rootless Podman socket Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): add OCI-backed image caching Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(test-guest): accelerate cached guest startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): address review feedback Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): verify OCI cache provenance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): harden cached guest reuse Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): refresh runtime setup state Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): support E2E runner inputs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test-guest): harden runner and OCI reuse Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(test-guest): prepare Podman E2E artifacts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(e2e): add host and test VM runner Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): add VM-backed Podman shutdown suite Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): use renamed test guest app Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(e2e): rename runner task Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): make guest smoke examples portable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): remove Podman shutdown suite Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(e2e): use test guest Podman setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(e2e): run Rust suites directly Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(e2e): simplify suite runner Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(e2e): isolate runner runtime state Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): prepare Podman VM runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(e2e): speed up cached guest startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(e2e): streamline guest gateway startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(server): isolate gateway listener context Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(server): preserve gateway listener binding Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(server): preserve covered listener callback scope Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(server): reuse gateway listener spec Signed-off-by: Evan Lezar <elezar@nvidia.com> * ci(e2e): pin rootless podman packages Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Nave Cohen <nave@monday.com>
NaveCohenMonday
force-pushed
the
2517-notify-atomic-policy-watchers/nc
branch
from
July 30, 2026 10:18
b9917c4 to
2870236
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Notify sandbox watchers after an atomic
UpdateConfigpolicy revision commits.Without this notification, live
WatchSandbox(follow_status=true)consumers cancontinue showing stale sandbox state until another event occurs. Sandbox
supervisors poll
GetSandboxConfigindependently and are not affected by thisnotification gap.
Related Issue
Fixes NVIDIA#2517
Changes
put_policy_revision_atomicsucceeds.one committed revision emits exactly one watcher notification.
notify observers.
Testing
CARGO_INCREMENTAL=0 mise run pre-commitpassesFocused test:
Checklist